{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a RAG Pipeline over IKEA Product Instruction Manuals\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_parse/blob/main/examples/multimodal/product_manual_rag.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cookbook shows how to use LlamaParse and OpenAI's multimodal models to query over IKEA instruction manual PDFs, which mainly contain images and diagrams to show how one can assemble the product.\n",
    "\n",
    "LlamaParse and multimodal LLMs can interpret these diagrams and translate them into textual instructions. With textual assistance, confusing visual instructions within the IKEA product manuals can be made easier to understand and interpret. Additionally, textual instructions can be helpful for those who are visually impaired."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install and Setup\n",
    "\n",
    "Install LlamaIndex, download the data, and apply `nest_asyncio`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index llama-parse llama-index-multi-modal-llms-openai git+https://github.com/openai/CLIP.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://github.com/user-attachments/files/16461058/data.zip -O data.zip\n",
    "!unzip -o data.zip\n",
    "!rm data.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up your OpenAI and LlamaCloud keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"<Your OpenAI API Key>\"\n",
    "os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"<Your LlamaCloud API Key>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up LlamaParse. We will parse the PDF files into markdown and use the GPT-4o multimodal model to parse the PDFs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load data from the parser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_parse import LlamaParse\n",
    "\n",
    "parser = LlamaParse(\n",
    "    result_type=\"markdown\",\n",
    "    parsing_instruction=\"You are given IKEA assembly instruction manuals\",\n",
    "    use_vendor_multimodal_model=True,\n",
    "    vendor_multimodal_model_name=\"openai-gpt4o\",\n",
    "    show_progress=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_DIR = \"data\"\n",
    "\n",
    "\n",
    "def get_data_files(data_dir=DATA_DIR) -> list[str]:\n",
    "    files = []\n",
    "    for f in os.listdir(data_dir):\n",
    "        fname = os.path.join(data_dir, f)\n",
    "        if os.path.isfile(fname):\n",
    "            files.append(fname)\n",
    "    return files\n",
    "\n",
    "\n",
    "files = get_data_files()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load data into docs, and save images from PDFs into `data_images` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "md_json_objs = parser.get_json_result(files)\n",
    "md_json_list = md_json_objs[0][\"pages\"]\n",
    "image_dicts = parser.get_images(md_json_objs, download_path=\"data_images\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create helper functions to create a list of `TextNode`s from the markdown tables to feed into the `VectorStoreIndex`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from pathlib import Path\n",
    "import typing as t\n",
    "from llama_index.core.schema import TextNode\n",
    "\n",
    "\n",
    "def get_page_number(file_name):\n",
    "    \"\"\"Gets page number of images using regex on file names\"\"\"\n",
    "    match = re.search(r\"-page-(\\d+)\\.jpg$\", str(file_name))\n",
    "    if match:\n",
    "        return int(match.group(1))\n",
    "    return 0\n",
    "\n",
    "\n",
    "def _get_sorted_image_files(image_dir):\n",
    "    \"\"\"Get image files sorted by page.\"\"\"\n",
    "    raw_files = [f for f in list(Path(image_dir).iterdir()) if f.is_file()]\n",
    "    sorted_files = sorted(raw_files, key=get_page_number)\n",
    "    return sorted_files\n",
    "\n",
    "\n",
    "def get_text_nodes(json_dicts, image_dir) -> t.List[TextNode]:\n",
    "    \"\"\"Creates nodes from json + images\"\"\"\n",
    "\n",
    "    nodes = []\n",
    "\n",
    "    docs = [doc[\"md\"] for doc in json_dicts]  # extract text\n",
    "    image_files = _get_sorted_image_files(image_dir)  # extract images\n",
    "\n",
    "    for idx, doc in enumerate(docs):\n",
    "        # adds both a text node and the corresponding image node (jpg of the page) for each page\n",
    "        node = TextNode(\n",
    "            text=doc,\n",
    "            metadata={\"image_path\": str(image_files[idx]), \"page_num\": idx + 1},\n",
    "        )\n",
    "        nodes.append(node)\n",
    "\n",
    "    return nodes\n",
    "\n",
    "\n",
    "text_nodes = get_text_nodes(md_json_list, \"data_images\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Index the documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import (\n",
    "    VectorStoreIndex,\n",
    "    StorageContext,\n",
    "    load_index_from_storage,\n",
    "    Settings,\n",
    ")\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding\n",
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "embed_model = OpenAIEmbedding(model=\"text-embedding-3-large\")\n",
    "llm = OpenAI(\"gpt-4o\")\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "\n",
    "if not os.path.exists(\"storage_ikea\"):\n",
    "    index = VectorStoreIndex(text_nodes, embed_model=embed_model)\n",
    "    index.storage_context.persist(persist_dir=\"./storage_ikea\")\n",
    "else:\n",
    "    ctx = StorageContext.from_defaults(persist_dir=\"./storage_ikea\")\n",
    "    index = load_index_from_storage(ctx)\n",
    "\n",
    "retriever = index.as_retriever()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a custom query engine that uses GPT-4o's multimodal model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.query_engine import CustomQueryEngine\n",
    "from llama_index.core.retrievers import BaseRetriever\n",
    "from llama_index.multi_modal_llms.openai import OpenAIMultiModal\n",
    "from llama_index.core.schema import NodeWithScore, MetadataMode\n",
    "from llama_index.core.base.response.schema import Response\n",
    "from llama_index.core.prompts import PromptTemplate\n",
    "from llama_index.core.schema import ImageNode\n",
    "\n",
    "QA_PROMPT_TMPL = \"\"\"\\\n",
    "Below we give parsed text from slides in two different formats, as well as the image.\n",
    "\n",
    "We parse the text in both 'markdown' mode as well as 'raw text' mode. Markdown mode attempts \\\n",
    "to convert relevant diagrams into tables, whereas raw text tries to maintain the rough spatial \\\n",
    "layout of the text.\n",
    "\n",
    "Use the image information first and foremost. ONLY use the text/markdown information \n",
    "if you can't understand the image.\n",
    "\n",
    "---------------------\n",
    "{context_str}\n",
    "---------------------\n",
    "Given the context information and not prior knowledge, answer the query. Explain whether you got the answer\n",
    "from the parsed markdown or raw text or image, and if there's discrepancies, and your reasoning for the final answer.\n",
    "\n",
    "Query: {query_str}\n",
    "Answer: \"\"\"\n",
    "\n",
    "QA_PROMPT = PromptTemplate(QA_PROMPT_TMPL)\n",
    "\n",
    "gpt_4o_mm = OpenAIMultiModal(model=\"gpt-4o\", max_new_tokens=4096)\n",
    "\n",
    "\n",
    "class MultimodalQueryEngine(CustomQueryEngine):\n",
    "    qa_prompt: PromptTemplate\n",
    "    retriever: BaseRetriever\n",
    "    multi_modal_llm: OpenAIMultiModal\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        qa_prompt: PromptTemplate,\n",
    "        retriever: BaseRetriever,\n",
    "        multi_modal_llm: OpenAIMultiModal,\n",
    "    ):\n",
    "        super().__init__(\n",
    "            qa_prompt=qa_prompt, retriever=retriever, multi_modal_llm=multi_modal_llm\n",
    "        )\n",
    "\n",
    "    def custom_query(self, query_str: str):\n",
    "        # retrieve most relevant nodes\n",
    "        nodes = self.retriever.retrieve(query_str)\n",
    "\n",
    "        # create image nodes from the image associated with those nodes\n",
    "        image_nodes = [\n",
    "            NodeWithScore(node=ImageNode(image_path=n.node.metadata[\"image_path\"]))\n",
    "            for n in nodes\n",
    "        ]\n",
    "\n",
    "        # create context string from parsed markdown text\n",
    "        ctx_str = \"\\n\\n\".join(\n",
    "            [r.node.get_content(metadata_mode=MetadataMode.LLM) for r in nodes]\n",
    "        )\n",
    "        # prompt for the LLM\n",
    "        fmt_prompt = self.qa_prompt.format(context_str=ctx_str, query_str=query_str)\n",
    "\n",
    "        # use the multimodal LLM to interpret images and generate a response to the prompt\n",
    "        llm_repsonse = self.multi_modal_llm.complete(\n",
    "            prompt=fmt_prompt,\n",
    "            image_documents=[image_node.node for image_node in image_nodes],\n",
    "        )\n",
    "        return Response(\n",
    "            response=str(llm_repsonse),\n",
    "            source_nodes=nodes,\n",
    "            metadata={\"text_nodes\": text_nodes, \"image_nodes\": image_nodes},\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a query engine instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = MultimodalQueryEngine(\n",
    "    qa_prompt=QA_PROMPT,\n",
    "    retriever=index.as_retriever(similarity_top_k=9),\n",
    "    multi_modal_llm=gpt_4o_mm,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Example Queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The query asks about the parts included in the Uppspel, but the provided images and parsed text do not contain any information about the Uppspel. Instead, they contain information about other IKEA products such as SMÅGÖRA, FREDDE, and TUFFING.\n",
       "\n",
       "Therefore, based on the provided images and parsed text, I cannot determine the parts included in the Uppspel. The answer cannot be derived from the given information."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, Markdown\n",
    "\n",
    "response = query_engine.query(\"What parts are included in the Uppspel?\")\n",
    "display(Markdown(str(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The Tuffing is a bunk bed frame with a minimalist design, featuring a metal frame and safety rails on the top bunk. The image provided shows the Tuffing bunk bed with a ladder for access to the top bunk and a simple, sturdy construction.\n",
       "\n",
       "I got the answer from the image provided. The image clearly shows the design and structure of the Tuffing bunk bed. There were no discrepancies between the parsed markdown or raw text and the image. The image was the primary source for understanding what the Tuffing looks like."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_engine.query(\"What does the Tuffing look like?\")\n",
    "display(Markdown(str(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The query asks for step 4 of assembling the Nordli. Based on the provided information, step 4 is described in the parsed text as follows:\n",
       "\n",
       "**Step 4:**\n",
       "- Insert the provided tool into the hole as shown.\n",
       "- Ensure the structure is properly aligned and secure.\n",
       "- Push down firmly to lock the structure in place.\n",
       "\n",
       "This information was derived from the parsed text, as the image provided does not contain step-by-step instructions for the Nordli assembly. There are no discrepancies between the parsed markdown and raw text for this step."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_engine.query(\"What is step 4 of assembling the Nordli?\")\n",
    "display(Markdown(str(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "If you're confused with reading the manual, you should contact IKEA customer service for assistance. This information is derived from the image on page 2, which shows a person with a question mark next to an IKEA box and another person making a phone call to IKEA. This visual cue indicates that contacting IKEA customer service is the recommended action if you need help."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_engine.query(\n",
    "    \"What should I do if I'm confused with reading the manual?\"\n",
    ")\n",
    "display(Markdown(str(response)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also create an agent around the query engine and chat with the agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.agent import FunctionCallingAgentWorker\n",
    "from llama_index.core.tools import QueryEngineTool\n",
    "\n",
    "query_engine_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=query_engine,\n",
    "    name=\"query_engine_tool\",\n",
    "    description=\"Useful for retrieving specific context from the data. Do NOT select if question asks for a summary of the data.\",\n",
    ")\n",
    "agent = FunctionCallingAgentWorker.from_tools(\n",
    "    [query_engine_tool], llm=llm, verbose=True\n",
    ").as_agent()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Give a step-by-step instruction guide on how to assemble the Smagora\n",
      "=== Calling Function ===\n",
      "Calling function: query_engine_tool with args: {\"input\": \"step-by-step instruction guide on how to assemble the Smagora\"}\n",
      "=== Function Output ===\n",
      "The step-by-step instruction guide on how to assemble the Smågåra crib is provided in the images. The images show detailed visual instructions for each step of the assembly process, including the tools required, the parts involved, and the specific actions to be taken.\n",
      "\n",
      "Here is a summary of the steps based on the images:\n",
      "\n",
      "1. **Tools Required**:\n",
      "   - Flathead screwdriver\n",
      "   - Phillips screwdriver\n",
      "   - Hammer\n",
      "\n",
      "2. **Preparation**:\n",
      "   - Do not assemble alone; assemble with a partner.\n",
      "   - Do not assemble on a hard surface; use a soft surface to avoid damage.\n",
      "   - If you have questions or need assistance, contact IKEA customer service.\n",
      "\n",
      "3. **Step 1**:\n",
      "   - Insert 12 screws into the designated holes on the frame.\n",
      "\n",
      "4. **Step 2**:\n",
      "   - Align the side panels with the headboard and footboard.\n",
      "   - Use 4 connectors and secure them with bolts and washers.\n",
      "   - Tighten using the provided tool.\n",
      "   - Carefully flip the structure as shown.\n",
      "\n",
      "5. **Step 3**:\n",
      "   - Use the provided Allen key to tighten the screws into the designated holes.\n",
      "   - Ensure the screws are properly aligned and tightened.\n",
      "   - Repeat this process for all four screws.\n",
      "   - Make sure the screws are flush with the surface.\n",
      "\n",
      "6. **Step 4**:\n",
      "   - Insert the provided tool into the hole as shown.\n",
      "   - Ensure the structure is properly aligned and secure.\n",
      "   - Push down firmly to lock the structure in place.\n",
      "\n",
      "7. **Step 5**:\n",
      "   - Insert 4 dowels into the designated holes on the board.\n",
      "\n",
      "8. **Step 6**:\n",
      "   - Align the board with the dowels and insert it into the corresponding slots on the frame.\n",
      "\n",
      "9. **Step 7**:\n",
      "   - Insert the top panel into the side panels.\n",
      "   - Use 4 screws to secure the top panel.\n",
      "   - Ensure the screws are properly aligned and tightened using the provided tool.\n",
      "\n",
      "10. **Step 8**:\n",
      "    - Carefully flip the assembled structure upright.\n",
      "    - Use 2 screws to secure the bottom panel.\n",
      "    - Tighten the screws with the provided tool.\n",
      "\n",
      "These steps are derived from the images provided, which offer a clear and detailed visual guide for assembling the Smågåra crib.\n",
      "=== LLM Response ===\n",
      "Here is a step-by-step instruction guide on how to assemble the Smågåra crib:\n",
      "\n",
      "### Tools Required:\n",
      "- Flathead screwdriver\n",
      "- Phillips screwdriver\n",
      "- Hammer\n",
      "- Allen key (provided in the package)\n",
      "\n",
      "### Preparation:\n",
      "- **Safety First**: Assemble with a partner to ensure safety and ease.\n",
      "- **Surface**: Assemble on a soft surface to avoid damaging the parts.\n",
      "- **Assistance**: If you have questions or need help, contact IKEA customer service.\n",
      "\n",
      "### Step-by-Step Assembly:\n",
      "\n",
      "#### Step 1: Insert Screws into the Frame\n",
      "1. Insert 12 screws into the designated holes on the frame.\n",
      "2. Ensure the screws are properly aligned.\n",
      "\n",
      "#### Step 2: Align and Secure Side Panels\n",
      "1. Align the side panels with the headboard and footboard.\n",
      "2. Use 4 connectors and secure them with bolts and washers.\n",
      "3. Tighten the bolts using the provided tool.\n",
      "4. Carefully flip the structure as shown in the instructions.\n",
      "\n",
      "#### Step 3: Tighten Screws\n",
      "1. Use the provided Allen key to tighten the screws into the designated holes.\n",
      "2. Ensure the screws are properly aligned and tightened.\n",
      "3. Repeat this process for all four screws.\n",
      "4. Make sure the screws are flush with the surface.\n",
      "\n",
      "#### Step 4: Lock the Structure\n",
      "1. Insert the provided tool into the hole as shown.\n",
      "2. Ensure the structure is properly aligned and secure.\n",
      "3. Push down firmly to lock the structure in place.\n",
      "\n",
      "#### Step 5: Insert Dowels\n",
      "1. Insert 4 dowels into the designated holes on the board.\n",
      "\n",
      "#### Step 6: Align and Insert the Board\n",
      "1. Align the board with the dowels.\n",
      "2. Insert the board into the corresponding slots on the frame.\n",
      "\n",
      "#### Step 7: Secure the Top Panel\n",
      "1. Insert the top panel into the side panels.\n",
      "2. Use 4 screws to secure the top panel.\n",
      "3. Ensure the screws are properly aligned and tightened using the provided tool.\n",
      "\n",
      "#### Step 8: Secure the Bottom Panel\n",
      "1. Carefully flip the assembled structure upright.\n",
      "2. Use 2 screws to secure the bottom panel.\n",
      "3. Tighten the screws with the provided tool.\n",
      "\n",
      "By following these steps, you should be able to assemble the Smågåra crib successfully. If you encounter any issues, refer to the visual instructions provided in the package or contact IKEA customer service for assistance.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "Here is a step-by-step instruction guide on how to assemble the Smågåra crib:\n",
       "\n",
       "### Tools Required:\n",
       "- Flathead screwdriver\n",
       "- Phillips screwdriver\n",
       "- Hammer\n",
       "- Allen key (provided in the package)\n",
       "\n",
       "### Preparation:\n",
       "- **Safety First**: Assemble with a partner to ensure safety and ease.\n",
       "- **Surface**: Assemble on a soft surface to avoid damaging the parts.\n",
       "- **Assistance**: If you have questions or need help, contact IKEA customer service.\n",
       "\n",
       "### Step-by-Step Assembly:\n",
       "\n",
       "#### Step 1: Insert Screws into the Frame\n",
       "1. Insert 12 screws into the designated holes on the frame.\n",
       "2. Ensure the screws are properly aligned.\n",
       "\n",
       "#### Step 2: Align and Secure Side Panels\n",
       "1. Align the side panels with the headboard and footboard.\n",
       "2. Use 4 connectors and secure them with bolts and washers.\n",
       "3. Tighten the bolts using the provided tool.\n",
       "4. Carefully flip the structure as shown in the instructions.\n",
       "\n",
       "#### Step 3: Tighten Screws\n",
       "1. Use the provided Allen key to tighten the screws into the designated holes.\n",
       "2. Ensure the screws are properly aligned and tightened.\n",
       "3. Repeat this process for all four screws.\n",
       "4. Make sure the screws are flush with the surface.\n",
       "\n",
       "#### Step 4: Lock the Structure\n",
       "1. Insert the provided tool into the hole as shown.\n",
       "2. Ensure the structure is properly aligned and secure.\n",
       "3. Push down firmly to lock the structure in place.\n",
       "\n",
       "#### Step 5: Insert Dowels\n",
       "1. Insert 4 dowels into the designated holes on the board.\n",
       "\n",
       "#### Step 6: Align and Insert the Board\n",
       "1. Align the board with the dowels.\n",
       "2. Insert the board into the corresponding slots on the frame.\n",
       "\n",
       "#### Step 7: Secure the Top Panel\n",
       "1. Insert the top panel into the side panels.\n",
       "2. Use 4 screws to secure the top panel.\n",
       "3. Ensure the screws are properly aligned and tightened using the provided tool.\n",
       "\n",
       "#### Step 8: Secure the Bottom Panel\n",
       "1. Carefully flip the assembled structure upright.\n",
       "2. Use 2 screws to secure the bottom panel.\n",
       "3. Tighten the screws with the provided tool.\n",
       "\n",
       "By following these steps, you should be able to assemble the Smågåra crib successfully. If you encounter any issues, refer to the visual instructions provided in the package or contact IKEA customer service for assistance."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = agent.chat(\n",
    "    \"Give a step-by-step instruction guide on how to assemble the Smagora\"\n",
    ")\n",
    "display(Markdown(str(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: How do I assemble the Fredde?\n",
      "=== Calling Function ===\n",
      "Calling function: query_engine_tool with args: {\"input\": \"step-by-step instruction guide on how to assemble the Fredde\"}\n",
      "=== Function Output ===\n",
      "The query asks for a step-by-step instruction guide on how to assemble the Fredde. However, based on the provided images and parsed text, there is no specific mention or visual representation of the Fredde assembly instructions. The images and text provided are related to other IKEA products such as Tuffing and Smågöra, but not Fredde.\n",
      "\n",
      "Therefore, I cannot provide the step-by-step instructions for assembling the Fredde from the given information. If you have the specific instructions for Fredde, please provide them, and I can assist you further.\n",
      "=== LLM Response ===\n",
      "It appears that the specific step-by-step instructions for assembling the Fredde desk are not available in the provided data. However, I can offer a general guide based on typical assembly procedures for IKEA furniture. For the most accurate and detailed instructions, please refer to the assembly manual that comes with the product.\n",
      "\n",
      "### General Assembly Guide for Fredde Desk:\n",
      "\n",
      "#### Tools Required:\n",
      "- Phillips screwdriver\n",
      "- Flathead screwdriver\n",
      "- Allen key (usually provided in the package)\n",
      "- Hammer (if needed for dowels)\n",
      "\n",
      "### Step-by-Step Assembly:\n",
      "\n",
      "#### Step 1: Unpack and Organize\n",
      "1. **Unpack** all the parts and hardware.\n",
      "2. **Organize** the parts by type and size to make the assembly process easier.\n",
      "\n",
      "#### Step 2: Assemble the Main Frame\n",
      "1. **Connect the Side Panels**: Attach the side panels to the back panel using screws and dowels as indicated in the manual.\n",
      "2. **Secure the Bottom Panel**: Attach the bottom panel to the side panels.\n",
      "\n",
      "#### Step 3: Attach the Shelves\n",
      "1. **Install the Lower Shelves**: Insert the lower shelves into the designated slots and secure them with screws.\n",
      "2. **Install the Upper Shelves**: Repeat the process for the upper shelves.\n",
      "\n",
      "#### Step 4: Attach the Desktop\n",
      "1. **Align the Desktop**: Place the desktop on top of the frame, ensuring it is properly aligned.\n",
      "2. **Secure the Desktop**: Use screws to secure the desktop to the frame.\n",
      "\n",
      "#### Step 5: Install Additional Features\n",
      "1. **Attach Monitor Shelf**: If the Fredde desk includes a monitor shelf, attach it to the back panel using screws.\n",
      "2. **Install Side Extensions**: Attach any side extensions or additional shelves as per the instructions.\n",
      "\n",
      "#### Step 6: Final Adjustments\n",
      "1. **Check Stability**: Ensure all screws are tightened and the desk is stable.\n",
      "2. **Adjust Height**: If the desk has adjustable height features, set it to the desired height.\n",
      "\n",
      "#### Step 7: Clean Up\n",
      "1. **Remove Packaging**: Dispose of any packaging materials.\n",
      "2. **Organize Tools**: Put away your tools and clean the workspace.\n",
      "\n",
      "For the most accurate and detailed instructions, please refer to the assembly manual that comes with the Fredde desk. If you encounter any issues, IKEA customer service can provide additional support.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "It appears that the specific step-by-step instructions for assembling the Fredde desk are not available in the provided data. However, I can offer a general guide based on typical assembly procedures for IKEA furniture. For the most accurate and detailed instructions, please refer to the assembly manual that comes with the product.\n",
       "\n",
       "### General Assembly Guide for Fredde Desk:\n",
       "\n",
       "#### Tools Required:\n",
       "- Phillips screwdriver\n",
       "- Flathead screwdriver\n",
       "- Allen key (usually provided in the package)\n",
       "- Hammer (if needed for dowels)\n",
       "\n",
       "### Step-by-Step Assembly:\n",
       "\n",
       "#### Step 1: Unpack and Organize\n",
       "1. **Unpack** all the parts and hardware.\n",
       "2. **Organize** the parts by type and size to make the assembly process easier.\n",
       "\n",
       "#### Step 2: Assemble the Main Frame\n",
       "1. **Connect the Side Panels**: Attach the side panels to the back panel using screws and dowels as indicated in the manual.\n",
       "2. **Secure the Bottom Panel**: Attach the bottom panel to the side panels.\n",
       "\n",
       "#### Step 3: Attach the Shelves\n",
       "1. **Install the Lower Shelves**: Insert the lower shelves into the designated slots and secure them with screws.\n",
       "2. **Install the Upper Shelves**: Repeat the process for the upper shelves.\n",
       "\n",
       "#### Step 4: Attach the Desktop\n",
       "1. **Align the Desktop**: Place the desktop on top of the frame, ensuring it is properly aligned.\n",
       "2. **Secure the Desktop**: Use screws to secure the desktop to the frame.\n",
       "\n",
       "#### Step 5: Install Additional Features\n",
       "1. **Attach Monitor Shelf**: If the Fredde desk includes a monitor shelf, attach it to the back panel using screws.\n",
       "2. **Install Side Extensions**: Attach any side extensions or additional shelves as per the instructions.\n",
       "\n",
       "#### Step 6: Final Adjustments\n",
       "1. **Check Stability**: Ensure all screws are tightened and the desk is stable.\n",
       "2. **Adjust Height**: If the desk has adjustable height features, set it to the desired height.\n",
       "\n",
       "#### Step 7: Clean Up\n",
       "1. **Remove Packaging**: Dispose of any packaging materials.\n",
       "2. **Organize Tools**: Put away your tools and clean the workspace.\n",
       "\n",
       "For the most accurate and detailed instructions, please refer to the assembly manual that comes with the Fredde desk. If you encounter any issues, IKEA customer service can provide additional support."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = agent.chat(\"How do I assemble the Fredde?\")\n",
    "display(Markdown(str(response)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama-parse-5ZmnAQ0r-py3.11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
